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The FSU-CAI Center Tech Memo Series is intended 
to provide communication to other colleagues and interested 
professionals who are actively utilizing computers" in their 
research. The rationale for the Tech Memo Series is three- 
fold. First, pilot studies that show great promise and wiDL 
eventuate in research reports can be given a quick distribu- 
tion. Secondly, speeches given at professional meetings can 
be distributed for broad review and reaction. Third, the 
Tech Memo Series provides for distribution of pre -publication 
copies of research and iinplementation studies that after 
proper technical review will ultimately be found in profes- 
sional journals. 

in terms of substance, these reports will be concise, 
descriptive, and exploratory in nature. While cast within a 
CAI research model, a number of the reports will deal with 
technical implementation topics related to computers and 
their language or operating systems. Thus, we here at FSIT 
trust this Tech Memo Series will serve a useful service and 
communication for other workers in the area of computers 
and education. Any comments to the authors can be forwarded 
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ABSTRACT 



Individualized instruction presents problems in measurement 
which challenge the conventional measurement paradigms. Taking into 
consideration the problems of item variance characteristics of CAI , 
idiosyncratic learning sequences, and lack of a model for effective- 
ness assessment, this paper reviews various measurement techniques 
used at the Florida State University CAI Center. The R and D strate- 
gies focus on two major goals: measurement providing information on 
priorities for revision within the CAI course materials, and measure- 
ment speaking directly to the effectiveness of the instructional pro- 
cess. Measurement techniques are related to three levels of course 
characteristics, (a) microframe, (b) concept segments within a CAI 
course, and (c) course effectiveness models. Foreseeable future trends 
are briefly discussed. 



MEASUREMENT TECHNIQUES FOR INDIVIDUALIZED INSTRUCTION IN CAI 
Duncan N. Hansen and Barbara F. Johnson 

I. Introduction 

Of the many challenges that individualized instruction poses to 
conventional measurement paradigms, the most demanding is the per- 
formance criterion orientation of computer-assisted instruction. That 
is, the goal of the CAI program is for all students to reach a speci- 
fied level of performance through a sequence of objectives or mile- 
stones embedded within a training course. This CAI goal plays havoc 
with the variance characteristics of both the instructional and test 
response items found within the data records of the individualized 
course (Hansen, Dick, & Lippert, 1968). A second challenge to conven- 
tional measurement is the differential and incomplete learning 
sequences. These idiosyncratic sequences limit the applicatiqn of many 
classical psychometric models and techniques. Still a third CAI chall- 
enge is posed by the attempts to assess the total effectiveness of the 
CAI course; a serious cost-effectiveness assessment would require a 
utility model which presently does not exist. Even with these many 
challenges, progress is being made in the creation of new measurement 
techniques appropriate for the CAI domain. 

This paper reviews various measurement techniques used by 
educational researchers at the CAI Center at Florida State University, 
The strategy of their approaches to the problems has focused on two 
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major goals: (a) measurement procedures that yield outcomes that provide 
insightful information regai^ding the priorities for revision within the 
CAI course materials; and (b) measurement outcomes that speak directly 
to the effectiveness of the instructional process. In order to gain 
some insight into the pursuit -of these two major goals, the measurement 
techniques can be related to three levels of course characteristics , 
namely: (a) procedures typically utilized at the CAI microframe level; 

(b) procedures used for CAI concept segments found within a course; and 

(c) course effectiveness models which focus on the appropriateness and 
benefits of CAI course outcomes. 

II. Microframe Measurement Techniques 

Microframe indices are the dependent measures collected during 
field tryouts of individualized learning materials , such as mean propor- 
tion of correct responses, latency, and subjective confidence. Without 
a doubt, probability of correct response, or error rate, has been the 
major item statistic looked at within CAI microframe outcomes. These 
item statistics present problems for the instructional psychologist in 
that criterion levels are difficult to defend on either an emni'^ical or 
theoretical basis. For example, there is little evidence that responding 
with a 10% or less error rate leads to either superior terminal perform- 
ance or improved retention. Obviously, wide discrepancies in error rate 
can result in wide fluctuations in terminal performance; but then, 
exceedingly high error rates are interpreted as being indicative of 
inappropriate selection of subjects, or of poorly prepared CAI materials. 
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Even so, considering incorrect responses, especially in terms of identi- 
fying the type and kind of error, provides useful information for the 
CAI course revision process. 

CAI investigations have shown that a baseline plus iterative 
approach provides meaningful guidelines upon which to base CAI course 
revision. The primary measurement technique consists of establishing 
baseline performance via a series of daily or weekly tests administered 
in a conventional course setting. During the development of an under- 
graduate physics course (Hansen, Dick, & Lippert, 1968), these investi- 
gators administered physics tests according to concept sections to a 
substantial number of students enrolled in the traditional lecture- 
demonstration course. This provided a statistical data base for deciding 
whether the first CAI course was an improvement at the microframe level 
and for determining which CAI concept areas were most in need of revision. 
This iterative measurement approach is predicated on the assumption that 
error rates will be a function of both the type of presentation mode 
(textbooks, films, programmed instruction, homework problem sets), and the 
nature of the concepts, especially in a hierarchically organized course. 

Table 1 presents the outcome for the first revision of the physics 
materials. The far right column of the table shows the downward perform- 
ance trend for students taking the course in a conventional lecture 
demonstration. For revision, the CAI curriculum developers separated out 
types of questions associated with different presentation modes, namely, 
textbooks, films, and conceptual exercises. As can be seen in Table 1, 
performance on the textbook materials was relatively consistent and 
undoubtedly a direct function of immediate memory effects typically found 
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TABLE 1 

Mean Co:'.-"ect Pi^oport’ions on Fvrst Responses 
to Difte^ent Lesson Materia) Categories 



by Physics Topics for First Revision 



Concepts 


Textbook 


Fil ms 


Cor^ceptual 

Exercises 


Basel ine* 


Scientific Measure 


.698 


.611 


-586 


.591 


Optics and Light 


.733 


.675 


.673 


.578 


Force and Energy 


,706 


.547 


.666 


.483 


Electric «ty 


-.703 


.476 


-653 


.391 


Modern Physics 


.703 


,486 


.695 


.412 



*Datd co)«ected on prior student groups. 



in assessing textbool< comprehension. On the other hand, it is interest- 
ing to note the much wider fluctuation in microframe performance for the 
film presentations. As to the CAi conceptual exercises, computer- 
assisted insv^uct’on ?ed to superior critericn-level performance indicat- 
ing that the goa! of improved mastery of the m.ore difficult conceptual 
areas such as found in eiect'-icity and modern physics was achieved. From 
a methodological viewpoint, by summing croframe performance statistics 
over concepts, a better insight into relative CAI performance and the 
priorities; for revision was gained. This then led to a primary focus on 
improving the questions associated with the later physics film sections 
in the course and a secondary focus on providing more CAI conceptual 
exercises. The iterative revision process resulted in a final course 
performance of CAi students that was approximately 15% superior on the 
final exam In comparison with the conventional course students. 
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An additiona! measurement revision strategy Involves frequent 
pre- and posttest assessment on clusters of behavioral objectives (Lipe, 
1970), Typically, after analyzing difference scores for learning games, 
investigators rank them in order to identify the behavioral objectives 
of the course most in need of revision. The rationale for this ranking 
procedure is that the curriculum developer needs to be parsimonious in 
his effort and should focus on those behavioral objectives most critically 
in need of further development. Thus, a rank ordering of pre-posttest 
difference scores gives an index of the microframes most in need of 
revision. 

Latencies , Latencies on learning and testing materials have been 
utilized within measurement strategies for CAL Latencies on study frames 
(frames presenting the basic conceptual materials) appear to provide the 
best index of those concepts found most difficult for the students. For a 
junior high school science course run under CAl (Brown, Conlon, Dasenbrock, 
Kellogg, Teates, & Redfleld, 1970), a substantial inverse relationship 
appeared between study time and performance on criterion tests. That is, 
the negative correlation was of the magnitude r - -.82, Partial ing out 
the effect of differential lengths of passages still leaves a relatively 
high correlation, r = -,62, From a revision point of view, these latency 
values provided the basis upon which the basic presentation material was 
revised within this individualized CAI course. The approach was further 
substantiated by the finding of a moderate relatioip, r = .46, between 
reading comprehension and terminal course performance for the students in 
this junior high science CAI course. Thus, the capability for collecting 
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latencies on CAI presentation frames permits development of a performance 
index about relative reading or comprehension difficulty which can be 
combined with the item performance statistics for developing a more 
refined revision strategy. 

Confidence ratings . A recent study concerning a science learning 
game (Harvey, 1970) indicated that subjective confidence ratings on micro- 
frame science concepts yield results parallel to the terminal exam per- 
formance, that is, the students' ratings of their confidence in handling 
questions for specific science concepts were remarkable predictors of 
their terminal performance. In addition, pre- and posttreatment comparison 
of the confidence rating on concepts yielded significant positive gains 
similar to final test performance for the individualized approach which in 
turn was superior to the conventional lecture discussion condition. Thus, 
confidence ratings are one more index which can be employed within micro- 
frame measurement procedures. 

Redundancy identification . An additional microframe technique 
suggested by Holland (1965) for programmed instruction is that of blocking 
out materials to identify inefficient redundancies. Random procedures are 
used to block out sentences with a programmed frame. Similar to the cloze 
techniques, this procedure produces an index of the impact of specific 
instructional presentations. 

As an equivalent technique, CAI investigators (Brown, Hansen, 
Thomas, & King, 1970) have composed equivalent materials with differential 
redundancy levels. The study indicated that allowing students to self- 
select among redundancy levels leads to effective performance outcomes,, 
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That is, the better performing students consistently choose the more 
concise presentation and the students with the highest error rates tend 
to choose the most redundant materials. Though not a direct application 
of the blockout techniques, this redundancy self-selection procedure 
does provide performance indices by which to study presentation microframes 
as opposed to test microframes. Unfortunately, very little empirical 
work has been performed utilizing these redundancy techniques due to the 
additional materials preparation requirement, 

III. Conceptual Measurement Techniques 

The term "conceptual segments" refers to the grouping of concepts 
to form specified CAI learning sessions- For example, in the CAI physics 
course, the concepts relating to light would be a CAI concept segment. 

The basic measurement approach to these larger CAI units has been the 
attempt to develop quantitative learning models,. Extensive effort has 
been given to developing finite state models and applying them to begin- 
ning CAI mathematics problems (Suppes, Jerman, & Bryan, 1968) or sequencing 
of vocabulary words in initial reading (Atkinson & Wilson, 1969), In 
essence, the finite state models attempt to define a series of learned and 
unlearned states and to specify the transition probabilities so as to 
maintain a record of the current learning state of a student for a given 
concept. Given the history status, the investigator can decide between 
the need to continue presenting materials, or the need for review. 
Unfortunately, the results of experimentation using this type of mathe- 
matical model have been far from promising; alternative decision rules 
appear to lead to inconsequential differences between CAI optimization 
groups and nonoptimization groups. 
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Linear regression models - An aUernat'Ve approach consists of 
using a linear regression mode) to keep t-'ack of the nr'croframe indices 
referred to above, an/ dynamically predict terminal performance levels. 

The measurement techniques within this linear regression approach con- 
sist of a two-phased empirical development, First, substantial numbers 
of students are linearly directed through all the CAI material. As many 
dependent measures as possible are assessed, and then are regressed against 
terminal performance levels in order to establish relationships and 
associated Beta weights. For the second phase, the *inear regression model 
is dynamically employed to predict or identify all failure cause, and 
CAI remediation is then applied, 

In recent experiment performed in the FSU CAI laboratory (Rivers, 
1971), this methodology was employed. Init!ai’y, 35 students were 
linearly taken through a CAI program wh'ch used scientific concepts for 
relating heart failure and EKG drawings Next, the CAI course was 
segmented into nine concept areas, and the dependent measures of probability 
of correct response, mean latency on subcriterion Items, and trait and 
state anxiety indices were regressed on terminal performance levels. For 
each of the nine concepts, a linear regression equation was prepared by 
which to predict an Individual's performance on the terminal test. The 
experiment was then repeated with four different groups, as follows: (a) 
a regression optimization group which received remediation and additional 
practice only if it was predicted that their performance was falling below 
a preestablished criterion level of 80% on the terminal test; (b) a total 
remediation control group, that is, a group that received all possible 
remediation; (c) a student selection group that could self-select remediation 
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if desired; and (d) a no-remed’at''on cont^'oT group Table 2 presents the 
mean final examination performance for these four groups. As can be 
observed, the optimization group performed bette*" than the other three 
groups. Moreover, the outcomes were ordered out In a sensibly appro- 
priate fashion; i.e,, the total remediation group performed better than 
the student selection of '^eview mate:^’als, or the no-remed1ation group. 



TABLE 2 

Mean Final Examination Outcomes for 
the Optimization Experiment 



Groups 


Mean 


Percentage Correct* 


Regression Optimization Group 


82.30 


65% 


Total Remediation Control 


77,65 


62% 


Student Selection 


65:45 


52% 


No Remediation Control 


61.45 


49% 



*0ut of 126 



Experiments of this type indicate how measurement-based individualization 
can be extended beyond the concepts of individual learning rates or 
remedial review conditional on embedded subcriterion tests. In essence, 
what is gained through a dynmic or continuously updated history record 
is a more substantial way of predicting performance and intervening with 
appropriate CAI learning materials. It is worth noting that this 
technique can be used within computer-managed instruction or programmed 
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instruction if sufficient automation is applied at various test points via 
dynamic procedures. These optimization models may hold promise for 
integrating the performance indices associated with microframe measure- 
ment techniques. 

IV. Course Assessment 

Effectiveness methodology typically compares a lecture demonstra- 
tion course with some individualized CAI course approach. The most common 
finding is that those students with lower entry performance levels typi- 
cally improve significantly more than their counterparts under conventional 
instruction. For example, Table 3, presenting results from the individual- 
ized science learning game (Harvey, 1970) s shows that the experimental and 
conventional control groups, being split at the median, were equivalent 
on the pretest science achievement measure. Not only did the most signifi- 
cant improvement occur in the lower half of the experimental group, but 
also this group was substantially superior on a concept specific criterion 
test that reflected concepts embedded in the individualized materials. 

As previously mentioned, the subjective confidence ratings improved 
equivalently with the performance. In addition to this, a short attitude 
scale indicated significantly positive shifts for the experimental group, 
a common finding in individualized CAI instruction. Thus, most findings 
(Majer, 1969; Hagerty, 1970; Lawler, 1971) tend to support the enhanced 
final performance, positive attitudes, and higher subjective confidence 
of individualized approaches to instruction. 

As the majority of students come to achieve higher levels of 
mastery via individualization, differential personality factors gain 
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TABLE 3 

Mean Outcomes for Experimental and Control 
Groups for the Individualized 
Science Learning Game 



I 




Experimental 


Conventional 


Lecture Group 




Low Entry 
Group 


High Entry 
Group 


Low Entry 
Group 


High Entry 
Group 


General Science 
Achi evement 

Pretest 


38.94 


52.94 


39.06 


53.94 


Posttest 


56.72 


59.44 


41.78 


57.70 


Criterion Science 
Test 

Pretest 


25.50 


28.22 


23.72 


29.17 


Posttest 


43.88 


44.33 


26.39 


31.00 


Science Concept 
Confidence 

Pretreatment 


8.78 


9.72 


9.50 


10.67 


Posttreatment 


11.28 


13.28 


10.39 


11.44 


Attitude 

Pretreatment 


231.28 


281.61 


264.28 


256.48 


Posttreatment 


316.17 


339.28 


292.44 


264.17 



ascendancy in the instructional process. In the analysis of the CAI 
physics course, the investigators found that CAI students with a 
humanistic orientation, low orientation towards science and technology, 
and high needs for affiliation and trust tended to gain the most from 
the individualized CAI course (Majer, 1969). In comparison, a subgroup 
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of students in the conventional lecture session with a personality 
pattern just the reverse, i.e., theoretical ly oriented, high values for 
science, and autonomous tendenceis, was the high performing group. 

This type of personality investigation, while still in its beginning 
states, has implications for appropriate selection and assignment pro- 
cedures within individualized CAI treatments. The need is evident for 
more research in the affective domain as individualization receives a 
wider dissemination throughout all levels of education. 

Simulation . Simulation techniques for total course assessment 
can be established to relate course processes and outcomes. In a recent 
simulation study. King (1970) created a model of an individualized teacher 
training curriculum, and found that by using the variables of social 
extroversion (which was negatively correlated (r - -.70) with learning 
time) and cumulative grade point average (r = .36 with learning time), 
the mean learning time could be reduced by 40% by selecting on these 
cognitive and affective variables. The modeling of an individualized CAI 
courseng simulation techniques indicated, in King's (1970) study, 
that it was difficult to shift performance levels due to the high criterion 
levels observed but that learning time could be manipulated via selection 
procedures. This simulated finding is highly consistent with empirical 
findings for individualized courses and illustrates how simulation can aid 
in course revision and improved effectiveness. 

Cost assessment . Individualized instruction systems, especially 
those utilizing computers, have a higher cost per instructional hour. 
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Savings in learning time and increases in performance levels are counter- 
balanced by increased fiscal costs. Utility theory can be employed to 
related these above factors. Although this is an extremely detailed 
methodology, the approach can be characterized briefly: one attempts to 
calculate all potential losses by taking the probability of the risk 
which is the difference between perfect performance and actual performance 
and multiply this by the actual cost per instructional hour. These 
calculations allow for combining outcome performance level with actual 
instructional costs. 

Most recent analyses of this type indicate that the loss coef- 
ficients for individualized instruction tend to converge on the costs 
for conventional courses due to labor factors. Moi^eover, computer- 
managed instruction promises to offer an approach that is significantly 
cheaper with excellent learng outcomes. For example, in a number of 
courses running at the FSU-CAI Center using CMI (Hagerty, 1970; Lawler, 
1971; Dick & Gallagher, 1971), the costs tend to be approximately 60 
cents per hour. It should be noted that the computer contact time 
represents approximately 12% of the course contact involvement. But 
still, there is a great need for more effectiveness models for relating 
the current and future outcomes of various individualized approaches so 
that economical forecasts of their associated costs and outcome potential 
can be made. Unfortunately, the development of these types of models 
appears to be exceedingly complex and difficult. 

V. Future 

In considering future development of measurement techniques for 
individualized instruction, a number of trends are foreseeable. First, 
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there is a need for more extensive conventiona' evaluation, especially 
dealing with the topics of review and retention. Moreover, , it has been 
recommended that additional incidental measures, beyond at of attitude 
alone, be considered in evaluating an individualized course. Such 
factors as attendance, commitment to the importance of the curriculum, 
and career development are being recommended tor consideration. Finally, 
it is appropriate to recognize the need for models that relate the 
learning process to personality processes (Leher issey , 1971), since 
these affective variables become more important in criterion-oriented 
instruction. Primarily, linear regression models will be employed to 
perform these investigations, but it is hoped that the use of simulation 
techniques will become more frequent, since they have a great potential 
for increased sophistication. The benefit of simulation rests in the 
identification of the potential application of individualizing procedures 
such as appropriate selection and assignment of media treatment so as to 
optimize the potential learning outcomes. Thus, researchers can anticipate 
that simulation models will become increasingly prominent in attempting 
to relate the specific empirical outcomes of a given individualized course 
to its potential application in the broader context of a training system. 
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